Intelligent system and method for assessing structural damage using aerial imagery

ABSTRACT

A system and method for capturing and automated processing aerial images of structures to assess structural damage is disclosed. The system comprises a computing system used to obtain images of selected locations associated with known structures at different points in time before and after a natural disaster. The images are used to automatically create three-dimensional models that are used to detect the specific portions of a structure that may be damaged as well as the spatial extent of that damage. In addition, the imagery can be fed into a damage classifier that automatically classifies the degree of damage and generates accurate estimates of repair costs. The system and method may be deployed to quickly assess damage of structures in a disaster area and provide reports of the damage to homeowners and/or insurers.

TECHNICAL FIELD

The present disclosure generally relates to structural damageassessments, and in particular to a system and method for assessingdamage done to structures using aerial imagery, machine learningtechniques, and 3D modeling.

BACKGROUND

Following disasters such as floods, hurricanes, fires, and tornadoes,entities that insure properties in the disaster area may need to surveythe area in order to assess any damage that has impacted one or moreinsured properties. For large scale disaster areas, aerial imagery maybe used to assess damage. Specifically, an aerial vehicle may fly overthe disaster area collecting continuous images that may later becombined into a single orthomosaic image. These images can be used toidentify generally whether a structure has been damaged. However,obtaining further information regarding the extent of such damage hasremained a time-consuming and resource intensive task, typicallyrequiring a human agent to visit the structure in person to more closelyexamine the effects of the disaster. Even when such examinations arenecessary, local environmental conditions following a disaster canprevent access to the structure for several days or weeks. This processcan lead to delays for homeowners and other insured entities inreceiving much needed relief or support. The ability to quickly andaccurately detect what portions of a structure have been damaged andcorrectly determining the degree of such damage, without the need foron-site manual inspections or other time-intensive tasks, is highlydesirable.

There is a need in the art for a system and method that addresses theshortcomings discussed above.

SUMMARY

In one aspect, a method of improving the accuracy of a damage assessmentbased on aerial imagery is disclosed. The method includes a first stepof obtaining a first set of imagery of a first property captured at afirst time, and a second step of automatically creating a firstthree-dimensional (3D) model of the first property based on the firstset of imagery. The method further includes a third step of obtaining asecond set of imagery of the first property captured at a second timesubsequent to the first time, and a fourth step of automaticallycreating a second 3D model of the first property based on the second setof imagery. In addition, the method includes a fifth step ofautomatically comparing the first 3D model and the second 3D model usinga machine learning model to detect damage on the first property that hasoccurred in the interval between the first time and the second time, anda sixth step of generating and presenting a damage report based on thecomparison of the first 3D model and the second 3D model.

In another aspect, an alternate method of improving the accuracy of adamage assessment based on aerial imagery is disclosed. The methodincludes a first step of obtaining a first set of imagery of a firstproperty captured at a first time, and a second step of obtaining asecond set of imagery of the first property captured at a second timesubsequent to the first time, where a natural disaster has impacted thefirst property during the interval between the first time and the secondtime. In addition, the method includes a third step of feeding the firstset of imagery and the second set of imagery to a deep learning damageclassification model, and a fourth step of determining, via the deeplearning classification model, that a first portion of a first structurelocated on the first property has been damaged, and identifying varyingdamage magnitude levels associated with the first portion. Finally, themethod includes a fifth step of generating and presenting a first heatmap that visually represents the damage magnitude levels associated withthe first portion.

In another aspect, a system for improving the accuracy of a damageassessment based on aerial imagery includes a processor andmachine-readable media including instructions which, when executed bythe processor, cause the processor to obtain a first set of imagery of afirst property captured at a first time, and to automatically create afirst three-dimensional (3D) model of the first property based on thefirst set of imagery. The instructions further cause the processor toobtain a second set of imagery of the first property captured at asecond time subsequent to the first time, and to automatically create asecond 3D model of the first property based on the second set ofimagery. In addition, the instructions cause the processor toautomatically compare the first 3D model and the second 3D model using amachine learning model to detect damage on the first property that hasoccurred in the interval between the first time and the second time, andto generate and present a damage report based on the comparison of thefirst 3D model and the second 3D model.

Other systems, methods, features, and advantages of the disclosure willbe, or will become, apparent to one of ordinary skill in the art uponexamination of the following figures and detailed description. It isintended that all such additional systems, methods, features, andadvantages be included within this description and this summary, bewithin the scope of the disclosure, and be protected by the followingclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention can be better understood with reference to the followingdrawings and description. The components in the figures are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the invention. Moreover, in the figures, likereference numerals designate corresponding parts throughout thedifferent views.

FIG. 1A is a schematic view of a three-dimensional model created by adamage assessment system, according to an embodiment;

FIG. 1B is a schematic view of a visual dashboard depictingdamage-related metrics for a selected region;

FIGS. 2A-2D are schematic views of a damage assessment flow process,according to an embodiment;

FIGS. 3A and 3B are schematic views of a process for developing andimplementing a model, and processing images, according to an embodiment;

FIGS. 4A and 4B are high-level views of a scenario in which an aerialvehicle collects image data over two periods of time, according to anembodiment;

FIG. 5 is a schematic view of a user interface presenting a map withmultiple structures identified as being damaged following a naturaldisaster, according to an embodiment;

FIG. 6 is an example of a user interface for viewing details ofstructures identified as being damaged by the damage assessment system,according to an embodiment;

FIGS. 7A and 7B are examples of two user interfaces for applying filtersin order to select specific structures identified as being damaged bythe damage assessment system, according to an embodiment;

FIGS. 8A and 8B are schematic views of two three-dimensional modelscreated by a damage assessment system based on imagery captured at twodifferent times, according to an embodiment;

FIG. 9 is a schematic view of the damage assessment system automaticallydetecting damaged portions in a structure based on the two models ofFIGS. 8A and 8B, according to an embodiment;

FIG. 10 is a schematic view of a process of employing a deep learningmodel to generate heat maps representing varying magnitudes of damage toa property, according to an embodiment; and

FIG. 11 a flow chart depicting a process of improving the accuracy of adamage assessment based on aerial imagery, according to an embodiment.

DESCRIPTION OF THE EMBODIMENTS

The embodiments provide a system and method for improving and automatingthe identification and inspection of buildings or other structures usingaerial imagery. Conventionally, aerial images are collected andassembled into a mosaic image, usually by a third party. The mosaicimage is then be analyzed to find buildings that may be damaged eitherimmediately following an acute event such as a natural disaster or moregradually over time. Using current methods, a building may be broadlyidentified as being either “damaged” or “not damaged” based on theimages. These conventional methods are further limited to detectingexternal damage to a building.

In contrast, the proposed system leverages machine learning to preciselyidentify the locations of buildings within a larger mosaic image, and toaccurately determine which portions of the building are damaged.Furthermore, the system can infer damage to internal structures, as wellas external structures. The disclosed embodiments can provide criticalinsights to insurance companies and other parties about specificportions of the building that were damaged and the extent of damage ateach portion, allowing for a more accurate estimate of rebuilding costs.The system implements a deep learning model to estimate damages acrossdifferent portions of the building, as well as a machine learning modelto detect potential damage to the sides of the building. In addition, a3D model of the building constructed using both nadir and oblique aerialimages is used to determine the dimensions for various portions of thebuilding. Information from each of these models enables the system toreadily pinpoint which portions of a structure have been damaged,including both external and internal components of the structure, aswell as the exact dimensions of the damaged regions.

For purposes of clarity, an overview of one embodiment of the proposedsystems and methods is illustrated with reference to FIGS. 1A and 1B. InFIGS. 1A and 1B, an embodiment of a smart damage assessment system(“system”) 100 implemented via a computing device 120 is depicted. Indifferent embodiments, the system 100 is configured to receive aerialimagery-based image data 130 and develop a three-dimensional model 134of a building, such as of a residence 102. As used herein, the term“building” or “structure” can refer to any kind of building, such as ahome, or other residential building, a shed, barn, a commercial buildingor any other related structures. A building typically can include aroof, room, walls, support structures, windows, or other features.

In some embodiments, the image data 130 can be used to generatedifferent models over time, allowing for a comprehensive and intelligentcomparison of the structure between a first point in time (e.g.,pre-disaster) and a second point in time (e.g., post-disaster). In otherwords, imaging data is used to build multiple 3D models of the structurewith sufficient precision to detect deviations in the structure overtime.

In FIG. 1A, the residence 102 has been exposed to a natural disaster,and as a result was heavily damaged. In this case, the residence 102 canbe seen to have suffered structural damage 132 as well as exposure tonearby fallen debris 136. The system 100 may detect the differences inthe appearance of the residence 102 and, with reference to output fromartificial intelligence-based machine learning models, determine boththe specific areas of the residence 102 that have been damaged and theextent of such damage. This assessment encompasses the entirety of thestructure of residence 102 by employing highly accurate 3D models of thestructure that have been generated based on image data 130 captured overmultiple different angles and spatial views and across different periodsof time.

In different embodiments, the system 100 can process the information andprovide an end-user with a variety of interactive tools to monitor andmanage the damage data. As one non-limiting example, FIG. 1B depicts auser dashboard 150 that can be provided to end-users to quickly conveyinformation and observations about a larger group of properties in theregion. The dashboard 150 in FIG. 1B presents an analytics overview forthe region in which residence 102 of FIG. 1A is located, and includes aninteractive header region 160 including a plurality of viewing options(e.g., “Event Summary”, “FAQs”, “Layer List”, etc.) and a metricssummary 170 with a legend 180 which can present a pictorial summary ofvarious aspects of the processed data. In other embodiments, the system100 can present various metrics and reports for individual structures aswell as for larger groups of structures in a region. In someembodiments, some or all of these reports can be presented in graphicalform, such as bar graphs, line graphs, pie charts, cartesian graphs,percent complete indicators, etc. In this case, a pie chart represents aspecific portion or feature of the structure (e.g., the roof, walls,solar panels, windows, etc.) that has suffered different degrees ofdamage, and the degree and amount or proportion of damage present inthat portion of the structure. For example, a rating of “Heavy Damage”,“Partial Damage”, “Low Damage”, and “No Damage” is used in FIG. 1B. Inother embodiments, other scales may be used, including numerical values,color coding, etc. As will be discussed below, system 100 can be used toautomatically detect and identify damage within the context of thespecific dimensions of the building as well as generate highly accurateestimates of the spatial extent of such damage. This type of datacollection, analysis, and visualization can provide essentialinformation to insurers and expedite the processing of claims forinsured members.

Referring now to FIGS. 2A-2D, an overview of an embodiment of data flowfor implementing the systems and methods disclosed herein is depicted.The data flow represents a process for capturing and processing imagesof structures covered by policies to improve the speed of assessingstructural damage and facilitating claim processing following adisaster. In different embodiments, some of the steps may be performedby ground-based computing system(s) and some of the steps may beperformed by aerial system(s).

For ease of presentation, the data flow is divided into fourinterconnected segments, including a first segment 202 (FIG. 2A), asecond segment 204 (FIG. 2B), a third segment 206 (FIG. 2C), and afourth segment 208 (FIG. 2D). The data flow can be understood to beginin FIG. 2A, where an aerial image capture service (AICS) such asGeospatial Intelligence Center (GIC) or other image collection entityinitiates an aerial imagery capture session for a target region at aninitial step 210. AICSs give insurers and associated damage assessmentsystems the ability to search an address and view before and afteraerials images of properties within the impacted area. Suchhigh-resolution aerial imagery provides insurers with vital informationto better serve policyholders, speed up the claims resolution process,and aid in improved fraud detection.

In order to appropriately organize and structure the image data, theimage collection is linked or otherwise associated with specificlocation data during a geocode stage 292. In a first step 212 of thegeocode stage 292, the system makes reference to a policy database forthe requesting insurer or party to identify and retrieve informationabout which structures are currently insured (“in-force policies inregion” or structures with “policies in force” (PIF)) or structures thatare otherwise relevant to the end-user and located within the targetregion. The address(es) of the relevant PIF structures are obtained in asecond step 214. For example, upon learning that a disaster hasoccurred, an insurance company (or other party) may prepare a list ofstructures in the disaster area that are insured and thus require damageassessments to be performed.

In some embodiments, the flow can then follow two sub-processes. In thefirst sub-process, as shown at a third step 216 of FIG. 2A, spatialdetails for a first structure are obtained, including parcel boundariesand a centroid for the structure. As a general matter, each structuremay be associated with location information. As used herein, the term“location information” refers to any kind of information that can befind a geographic location for an object. Location information mayinclude latitude and longitude information. Location information couldalso comprise a street address. It may be appreciated that locationinformation provided in one format (for example, a street address for astructure) could be converted into another format (for example, alatitude and longitude position). When the location information isspecific to a structure (such as a house, office building, or any otherstructure) the term “structure location” may be used. By obtaining alist of structure locations, the system can capture images of areas thatinclude the structure locations. For example, an airplane may fly overthe disaster area and photograph areas according to the provided list ofstructure locations.

This process is repeated in a fourth step 218 until the boundaries andcentroids for all relevant structures in the target region have beenacquired. This geospatial information is appended in a fifth step 220 toa file or other data structure directed to the designated PIF structure.The first sub-process (labeled as A-path) can in different embodimentsapply or be performed by a virtual survey tool such as CoreLogic® oranother survey system that can be used to virtually assess propertyexposure, condition and features, as well as survey post-catastrophedamage based on aerial imagery.

In a second sub-process, as shown at a sixth step 220 of FIG. 2A, theaddress information is filtered (e.g., “Filter toAddr_Type=PointAddress”) for processing and use by the AICS system. Inthis case, point address refers to a locator code or name created forcommon addresses that contain a street number and street name. Thislocator role uses feature classes with polygon or point geometry as theprimary reference data. Thus, each feature in the primary reference datacorresponds to a single address. However, other filters may be appliedto the address information based on the requirements of the AICS systembeing used. In a seventh step 222, the latitude and longitude of thecentroid is determined for each point address, and this geospatialinformation is appended in the fifth step 220 to a file or other datastructure directed to the designated PIF structure. The secondsub-process (labeled as B-path) can in different embodiments apply or besupplemented or performed by a set of deep learning generated buildingfootprints covering the target region such as Microsoft® BuildingFootprints. The footprints can be used for visualization using vectortile format or as a hosted feature layer to do analysis. This group ofdata for each PIF is stored locally within data files for the targetregion.

Furthermore, the collected data is conveyed for further processing to aMember Images Creation stage 294 (see FIG. 2B) and a Property DamageEstimator stage 260 (see FIG. 2D). Referring first to FIG. 2B, thecaptured aerial images 230 are reviewed to determine whether theyrepresent new imagery in an eighth step 232. If the system determinesthe imagery is not new, no further steps will be taken. If the systemdetermines the imagery is new, imagery footprint polygon for the imageryis obtained or created in a ninth step 234 using the spatial referenceof the mosaic dataset, which may be different from the source rasterdatasets. The imagery footprint polygon refers to the spatial area for agiven location search area, or a Well-Known Text (WKT) representation ofthe shape (footprint, geometry) that defines the location. The processmay be performed by a service such as but not limited to QuickBird (QB)or other services providing data sets that contain high-resolutionimagery and geospatial data for the target region. With QuickBird, eachorder is defined by an Order Polygon. An Order Polygon may contain aminimum of four vertices and a maximum of 1,000 vertices, consisting oflongitude/latitude (decimal degrees) geographic coordinates on theellipsoid. The minimum and maximum size for an order polygon depends onthe order type and the product selected. In a tenth step 236, the newpolygon in the footprint list of polygons is identified.

The new polygon, along with the output of FIG. 2A, are linked toidentify the PIF structure(s) located in the current event polygon in aneleventh step 238, and a bounding box is created around each of thecentroids for the structures in a twelfth step 240. A first sub-processthen clips the imagery layer to the structure level and both shares thisdata with a geoportal presentation layer stage 280 (see FIG. 2D) in athirteenth step 242. In a second sub-process, the imagery layer isclipped to the neighborhood level in a fourteenth step 244. In addition,in a fifteenth step 246, the imagery for the structure level and theneighborhood level (at least two images) is stored in association witheach property file. Finally, a PDF presenting the processed imagery forthe structure is created for each property in a sixteenth step 248,concluding the process segment of FIG. 2B.

Referring next to FIG. 2C, first notice of loss (FNOL) claim(s) arecreated in a FNOL Claims stage 296. A FNOL refers to the initial reportmade to an insurance provider following loss, theft, or damage of aninsured asset, and is normally the first step in the formal claimsprocess lifecycle. Thus, an automated process for generating such areport can be of great value to both the insurer and homeowner, as itgreatly expedites the process as a whole. In a seventeenth step 250, thesystem makes reference to an FNOL claims database and applies ageolocation filter (eighteenth step 252) and a timestamp filter(nineteenth step 254) to obtain the appropriate data. In a twentiethstep 256, an initial severity code, peril type code, policy number orreference, member number, and/or loss ID are assigned to the report.Finally, the FNOL claim report (e.g., a PDF or other documentationoutput) is generated in a twenty-first step 258 and sent to thegeoportal presentation layer stage 280 of FIG. 2D.

Referring now to FIG. 2D, two additional stages of the data flow processare depicted as fourth segment 208. In the property damage estimatorstage 260, output from FIG. 2A is received and imagery outside of thedilated false positives (FPs) or parcel boundary is masked in atwenty-second step 262. False positives (FP) are parcels which wereerroneously included by either machine or human experts. The image listis preprocessed for reference in a twenty-third step 264, and with theuse of trained damage estimation model(s) (see FIGS. 3A and 3B), adamage estimation model is produced in a twenty-fourth step 266 togenerate post-process predictions in a twenty-fifth step 268. Thisoutput is provided as input to the geoportal presentation layer stage280, along with the output from FIG. 2B. In a twenty-sixth step 282, thePIF that is in force as listed at the Special Direct Facility (SDF) isappended with damage prediction attributes from twenty-fifth step 268.Generally, buildings with certain insurance policies can be required toa monitoring agency to supervise the issuance of policies and the claimsprocess for policies in force on the property.

The map is then updated with damage SDF in twenty-seventh step 284 aswell as claim SDF (based on output from FIG. 2C) in a twenty-eighth step286. The web map is updated with the claim and damage map features in atwenty-ninth step 286, and finally the live link to the augmented,comprehensive map and report for each PIF structure is updated and madeavailable to system users in thirtieth step 290.

In different embodiments, the system can include provisions forgenerating highly accurate estimates of damage and repair costs. In FIG.3A, one embodiment of a flow process 300 for development andimplementation of a machine learning model for image processing isshown. The flow process 300 includes a first stage 302 (modeldevelopment) and a second stage 304 (model implementation). During thefirst stage 302, input 310 in the form of imagery and associatedannotations is provided to the system. The input 310 is cleansed andnormalized in a first step 312 and a CV dataset is created based on thisdata in a second step 314. In different embodiments, various imageprocessing algorithms and/or software may be used with captured imagedata. In one embodiment, the image processing algorithms performcompression, artifact correction, noise reduction, color corrections,geometric corrections, imager non-uniformity correction, etc., andvarious image processing enhancement operations on the image content.The algorithms can be implemented as software running on a processor,DSP processor, special purpose ASIC and/or FGPA's. The image processingalgorithms can also be a mixture of custom developed algorithms andlibraries. The image processing algorithms can further be arranged inany logical sequence, with potential changes in the sequence ofprocessing or parameters governing the processing determined by imagetype, computational requirements or outputs from other algorithms.

In some embodiments, image processing may also include machine learningtechniques that can be used to discriminate between features and toidentify objects, for example via image recognition and object detectionsoftware. Such techniques may also include machine vision algorithmsthat perform, among other operations, symbol and logo recognition,general shape recognition, as well as object classification. The machinevision algorithms may reside on a different system belonging to adifferent entity than the image processing algorithms or the applicationsoftware. The machine vision algorithms, which are applied to identifyan object in the digital image, may include computer vision algorithmssuch as image analysis algorithms that may use a feature detector or acombination of detectors. For example, texture detectors and edgedetectors known to those skilled in the art may be used. If bothspecific texture and specific edges are detected in a set of images,then an identification may be made. One non-limiting example of an edgedetection method includes the Canny™ algorithm available in computervision libraries such as Intel™ OpenCV. Texture detectors may use knownalgorithms such as texture detection algorithms provided by Matlab™.Some non-limiting examples of object detection algorithms include R-CNN,SPP, Fast R-CNN, Faster R-CNN, Feature Pyramid networks, RetinaNet(Focal loss), Yolo Framework—Yolo1, Yolo2, Yolo3, and SSD.

A cycle comprising a third step 316 in which the model is trained,validated, and a split dataset tested followed by a fourth step 318 inwhich the selected neural network (e.g., CNN, RNN, etc.) is trained andtuned based on the output of third step 316 then occurs. Morespecifically, machine learning techniques, such as deep learning thatincludes classification, clustering, and/or other techniques, areapplied to the CV dataset to develop the model(s). Such ML techniquesmay include, but are not limited to, techniques that employ deeplearning neural networks for pattern recognition within the image data,or to perform other types of analysis. For example, a neural networkand/or classification technique may be used to train a model that is aclassifier and that is useable to detect different types of damage. Somesuitable artificial intelligence software is available for public accessthrough open source AI platforms like Caffe, Torch and Theano whoprovide businesses access to powerful neural networks for processing oftheir information by AI techniques like deep learning, reinforcementlearning and logistic regression, as well as TensorFlow, OpenAI, andBigSur. All of these AI systems process enormous amounts of data; forexample, Caffe can process over 60 million images per day with a singleNVIDIA K40 GPU.

Moreover, in some implementations, the process may employ an estimationengine that uses ML techniques to generate repair cost estimateinformation. In some embodiments, such techniques may include supervisedand/or unsupervised ML techniques. In some implementations, theestimation engine may employ a ML-based model that is trained usingtraining data that includes prior cost estimates and actual costinformation. Accordingly, the estimation engine may be trained over timeto develop a more accurate cost estimate based on the previousdivergence between estimates and actual cost.

Once a model is selected in a fifth step 320, the trained model isadjusted to fit the intended deployment environment in a sixth step 322,and the resulting trained model 330 is delivered to the modelimplementation stage 304. In a seventh step 342, large-scale image data340 for PIF properties are used to identify and clip individual memberproperty imagery 344. Additional details regarding this process isprovided below with respect to FIG. 3B. The imagery 344, along withtrained model 330, is scored by a computer vision scoring service in aneighth step 348. The obtained estimated score is linked to the member'spolicy and claim information in a ninth step 348, and then published tothe interactive map in a tenth step 350.

For purposes of clarity, a flow process for image clip creation 360 isdepicted in FIG. 3B, where each member property 362 is identified andclipped, as noted in FIG. 3A. In a first step 364, a query is executedfor properties with PIF in the selected state or region, or otherlatitude/longitudinal SDS. In addition, in a second step 366 a query isexecuted to request an aerial coverage footprint. The query results offirst step 364 are filtered to obtain only those member properties withparcel geocode precision levels in a third step 368, and the resultsfrom third step 368 and second step 366 are further processed in afourth step 370 to obtain PIF properties with a point in the footprintpolygon, where each point represents a single specific location, such asan address, users location, or asset, and the footprint polygonrepresents closed and filled shapes such as state or country boundaries,parks or building footprints. They can include both holes andnon-overlapping geometries, and are used to calculate boundaries. In afifth step 372, the address for each property can be re-geocoded basedon the newest geocoder obtained, and in a sixth step 374 the aeriallayer is clipped at two zooms with PIFs that were re-geocoded at thepoint address, using for example x and y coordinates, for example byreference to the image server. Finally, the clip images are savedlocally in a seventh step 376.

For purposes of illustration, two schematic views of an aerial vehicle400 performing surveys of a target zone or region 436 are shown in thesequence of FIGS. 4A and 4B. In FIG. 4A, a first survey 450 is performedof a group of structures 430 located in the region 436 at a first time,while the structures are in a “whole” or undamaged state (i.e., prior tothe occurrence of a disaster for which the structures may be insured).As used herein, the term “aerial vehicle” refers to any kind of plane,helicopter, drone, or other flying vehicles. In this exemplaryembodiment, aerial vehicle 400, also referred to simply as vehicle 400,is a plane operated by a pilot. However, in other embodiments, vehicle400 could be remotely operated or programmed.

Vehicle 400 includes at least one camera for capturing images. Forpurposes of simplicity, in FIGS. 4A and 4B the vehicle 400 includes botha first camera 410 and a second camera 420, each oriented differently.However, in other embodiments, the vehicle 400 may employ a singlecamera that is configured to rotate or reorient its lens to captureimages at different angles. It is important that the imagery captureoccurs at varying angles in order to collect sufficient image data togenerate a 3D model. In some embodiments, the imagery can be obtainedusing both vertical imagery (nadir, 90 degrees) and oblique (45 degrees)imagery techniques. In general, vertical imagery 422 offers anapproximately straight-down aerial view of properties and locations,providing keen insight into rooftops and property surroundings, andat-a-glance situational awareness of large-scale catastrophe scenariossuch as city-wide flooding. In contrast, oblique imagery 412 provides anapproximately 45° perspective of properties and locations from all fourcardinal directions, allowing viewers to see and measure not only thetop of objects but the sides as well (e.g., the external walls andwindows of a building). In this example, first camera 410 is showncapturing oblique images, and second camera 420 is shown capturingvertical images. The two types of image techniques can be used tocollect multiple images of the same structure. In FIG. 4A, the vehicle400 obtains vertical and oblique imagery for both first house 432 andsecond house 434 in their pre-damaged or an initial or original state inorder to develop a reference 3D model that can be used to monitor anddetect changes in the condition of the structure over time.

In different embodiments, vehicle 400 may also include or be incommunication with additional systems to facilitate capturing,processing, and transmitting image information about one or more areas.For example, a damage detection system can comprise both a ground systemand an aerial system. The ground system includes provisions forgathering information about potentially damaged structures following adisaster that can be used to facilitate an image survey of the disasterarea. The ground system may also include provisions for processing imagedata and for communicating with various other systems. In addition, theaerial system (represented by vehicle 400) includes provisions forcapturing aerial images of one or more areas. The aerial system may alsoinclude provisions to determine precise locations for captured images,as well as for performing image processing.

Aerial system may comprise various systems and components that aredisposed within an aerial vehicle (such as aerial vehicle 400). As notedabove, the aerial system may include one or more cameras for capturingimages and information about a building structure. The camera maycomprise any kind of camera, including any kind of digital camera and/orrange imaging camera. Range imaging cameras include any type of devicethat can capture range information or range images corresponding to anobject in the viewing area of the camera. As used herein, “range images”provide a 2D array of values indicating a depth (or distancemeasurement). Some exemplary range imaging devices may includestructured-light 3D scanners and time-of-flight cameras. Using atime-of-flight camera, the system can capture range images of a scenethat can be used to build a 3D model of objects in the scene, such asbuilding structures.

The aerial system can also include a GPS receiver for receiving GPSinformation that can be used to determine a GPS location for the aerialvehicle. In some embodiments, the aerial system may also include sensorsfor measuring orientation, altitude, and/or acceleration. For example,an aerial system can include a gyroscope, an altimeter, and anaccelerometer. In some embodiments, the aerial system can include analtitude and heading reference system (AHRS). Using these devices, theorientation, heading, and height of the aerial vehicle (and ofcamera(s)) can be determined. This information, when used with a GPSlocation for the aerial vehicle, can be used to infer the location ofone or more points in an image taken from the aerial vehicle asdescribed in further detail below.

In different embodiments, aerial system can also include an imagecapture and processing system, also referred to simply as processingsystem. A processing system may be used to store, process, and transmitimage information. Additionally, in some cases, a processing system canreceive GPS or other coordinate information about one or more targetlocations. To facilitate these tasks, image capture and processingsystems may include one or more processors as well as memory. Memory canstore instructions for programs that facilitate storing, processing, andtransmitting image information.

Generally, the ground system comprises a computing system that caninclude, for example, a computer and a database. The computer mayfurther include one or more processors and memory. The computer could beany kind of computer such as a desktop computer, a laptop computer, aserver, or any other kind of computer with sufficient computingresources for performing tasks such as image classification. In someembodiments, the ground system can refer to a plurality ofinterconnected computing devices and/or cloud service repositoriesconfigured to connect over a network. Additionally, models or otherinformation could be stored in a separate model database of thecomputing system.

Furthermore, in some embodiments, computing system may also include adamage classifier. The damage classifier may be any program or algorithmthat is used to classify images according to the degree of damage thestructure has sustained. In some embodiments, damage classifier includesone or more machine learning models (see FIGS. 3A and 3B). In oneembodiment, damage classifier could include a convolutional neuralnetwork. In other embodiments, damage classifier could comprise anyother algorithm (or set of algorithms) from the field for machinelearning and/or machine vision.

The computing system may also incorporate provisions for displayingmodels of building structures to a user. In some embodiments, thecomputing system includes an augmented reality (AR) application, whichcan be used to superimpose a model of a building structure onto a scenecaptured by a photographic or other camera. In contrast to range imagingcamera, photographic cameras comprise sensors for capturing lightingand/or color information that can be used to build a 2D photographicimage of a scene. In some embodiments, augmented reality elements couldbe projected onto a display of a device of the ground system. In otherembodiments, a separate AR device, such as AR goggles, could be used todisplay AR information for a user. The computing system may furtherinclude components that facilitate creating and analyzing 3D models ofbuilding structures based on ranging image information. In someembodiments, the computing system may further include a model comparisonapplication.

In different embodiments, devices and components of the computing systemmay communicate over a network. Generally, a network could comprise anykind of network, such as but not limited to a Wide Area Network (WAN), aLocal Area Network (LAN), Wi-Fi network, Bluetooth or other PersonalArea Network, cellular network, as well as other kinds of networks. Itmay be appreciated that different devices could communicate usingdifferent networks and/or communication protocols. In other embodiments,a 3D modeling application could be configured to run on an end-userdevice, rather than on a separate system such as a server. In stillother embodiments, some components of a 3D modeling system could be runon a user device, while other components could be run on a server.

Both the ground system and aerial system can include communicationsystems. For example, the ground system can include a firstcommunication system and aerial system can include a secondcommunication system. These communication systems enable information tobe transmitted between the ground system and the aerial system via anetwork. Thus, the type of communication components used in eachcommunication system can be selected according to the type of networkused. In some cases, a cellular network could be used so that eachcommunication system includes a cellular radio or other component thatenables cellular communication. Using a cellular network may enableinformation to be exchanged while aerial system is in the air whereWi-Fi or other networks might be unavailable. In other cases, a networkcould comprise any kind of local area network and/or wide area network.In some cases, the network may be a Wi-Fi network. Alternatively, theground system and aerial system could be connected by wires, forexample, when aerial system is on the ground and near ground system.Furthermore, one or more components of the aerial system could bedisposed within a single computing device. Examples of computing devicesthat could be used include, but are not limited to: laptop computers,tablet computers, smartphones or other computing devices.

Referring now to FIG. 4B, the aerial system of vehicle 400 is depictedperforming a second survey 450 of the same group of structures 430located in the region 436 at a second, subsequent time, shortly afterregion 436 has experienced a natural disaster (e.g., tornado 490).During the second survey 450, the vehicle 400 again collects verticaland oblique imagery for both first house 432 and second house 434, nowin their damaged or altered state. The new imagery will be used todevelop an updated 3D model that can be used by the system to detectchanges in the condition of the structure resulting from the tornado490.

This type of collection or survey can be performed across large areas ofland to support the automated processing and generation of damageassessments for multiple structures in an affected region. In manycases, homes and other buildings in a city may have insurance policiesthat cover loss following a disaster such as a tornado. In FIG. 5, forpurposes of illustration, a map interface 500 is displayed in whichstructures located in an area 510 (here shown as Dallas, Tex.) arepresented. Based on techniques disclosed herein, the system is able toidentify specific buildings 520 that were (a) covered by a particularpolicy (have a PIF) and (b) have been affected to some degree by thedisaster. An end-user can modify or filter the results to generatealternate views. For example, the end-user may wish to only viewstructures that are classified as residences, or as businesses, or onlythose structures that were impacted above a selected damage threshold,or have a specific policy type. In some embodiments, the map interface500 can receive inputs to pan or magnify the view in order to accessadditional details for one or more structures. In one embodiment, aclick or selection of one of the structures can open a new windowpresenting additional details for that structure and/or trigger thepresentation of a new interface in which the end-user can interact withstructure-specific data and imagery.

Some non-limiting examples of such information panels and/or filteringoptions are shown with reference to FIGS. 6-7B. In differentembodiments, these interfaces are presented in conjunction withsatellite imagery or other depictions of the designated structures ortarget region. As a general matter, an “interface” may be understood torefer to a mechanism for communicating content through a clientapplication to an application user. In some examples, interfaces mayinclude pop-up windows that may be presented to a user via nativeapplication user interfaces (UIs), controls, actuatable interfaces,interactive buttons or other objects that may be shown to a user throughnative application UIs, as well as mechanisms that are native to aparticular application for presenting associated content with thosenative controls. In addition, the terms “actuation” or “actuation event”refers to an event (or specific sequence of events) associated with aparticular input or use of an application via an interface, which cantrigger a change in the display of the application. Furthermore, a“native control” refers to a mechanism for communicating content througha client application to an application user. For example, nativecontrols may include actuatable or selectable options or “buttons” thatmay be presented to a user via native application UIs, touch-screenaccess points, menus items, or other objects that may be shown to a userthrough native application UIs, segments of a larger interface, as wellas mechanisms that are native to a particular application for presentingassociated content with those native controls. Voice control can also beused to actuate options. The term “asset” refers to content that may bepresented in association with a native control in a native application.As some non-limiting examples, an asset may include text in anactuatable pop-up window, audio associated with the interactive click ofa button or other native application object, video associated with ateaching user interface, or other such information presentation.

FIG. 6 shows an example of a results panel 600 in which a “triage” list610 identifying a group of structures that the system has evaluated fordamage following a natural disaster. In this case, the system hasassessed 12,369 policy holder buildings, and assigned classificationsfor damage level, the probability of the damage level being accurate,and the member identification number. In FIG. 6, it can be seen that thelist 610 includes structures that have been classified as having“Heavy_DMG” with varying probabilities. The selected classification canbe applied as a result of damage classification model classifies eachstructure according to various levels of damage. As an example, thedamage classification model could assess structures as having “nodamage,” “minor damage,” “significant damage,” or “total loss”, oracross a more extensive metric, such as an assignment of a number (e.g.,between 0 and 10, 0 and 50, 0 and 100, or any other range) in which zerorepresents no damage and the maximum number represents catastrophicdamage. Of course, other classifications are possible. In addition toclassifying the structures according to levels of damage, a damageclassifier could also classify the amount of damage using other metricssuch as the cost of damage, the cost of payout, as well as otherpossible metrics.

FIGS. 7A and 7B depict examples of filtering options for a front-endsearch engine. In FIG. 7A, a first filter interface 700 includes aheader 702 with selectable filter categories (e.g., Filter the VirtualInspections on Roof Concerns”, “Filter on Coverage Type”, etc.) underwhich an additional submenu is selected (“Filter on Roof ConditionProbabilities”). In this case, some roof-related filters are offered,including a first filter option 710 (“Roof Damage Probability is GreaterThan _(——————)”), a second filter option 720 (“Tarp Probability isGreater Than _(——————)”), and a third filter 730 (“Roof DiscolorationProbability is Greater Than _(——————)”). For purposes of this example,the end-user has selected a high probability (0.98) for roof damage toview those houses with roofs that have been impacted, and a zeroprobability for tarp damage and roof discoloration to remove structureswith damaged tarps and/or roof discoloration from the search entirely(unless the same structure has both roof discoloration and one of theexcluded categories).

Similarly, in FIG. 7B, a second filter interface 750 includes a header752 (“Filter the Virtual Inspections on Property Concerns”) under whichan additional submenu 754 is selected (“Filter on Property ConditionProbabilities”). In this case, some property-related filters areoffered, including a fourth filter option 740 (“Yard Debris Probabilityis Greater Than _(——————)”), a fifth filter option 750 (“PoolProbability is Greater Than _(——————)”), a sixth filter option 760(“Solar Panel Probability is Greater Than _(——————)”), and a seventhfilter option 770 (“Trampoline Probability is Greater Than _(——————)”).For purposes of this example, the end-user has selected a highprobability (0.98) for yard debris to view those houses that haveincurred damage around their structure, as well as a high probability(0.98) of properties that include a trampoline, while selecting a zeroprobability for the presence of a pool and/or solar panel to remove suchproperties from the search entirely (unless the same structure has bothroof discoloration and one of the excluded categories).

It should be understood that the dashboard 150 of FIG. 1B and userinterfaces or panels of FIGS. 6-7B represent only one possible depictionof interfaces that may be offered to the end-user, and in otherembodiments, any variation in presentation style, options, menus, andgraphical depictions can be used, including interfaces customized by theend-user to display the desired information. A Settings option can alsobe displayed to allow the end-user to create or modify the account. Inaddition, a number of interface layer options may be provided. Forexample, other options can allow the user to switch to a landing pagethat presents a brief summary of the user's account or a log of theuser's previous activity.

In FIGS. 8A and 8B, an embodiment of a smart damage assessment system(“system”) 800 implemented via a computing device 802 is depicted. Indifferent embodiments, as described herein, the system 800 is configuredto receive aerial imagery-based image data for structures over differentperiods of time. In other words, the image data can be used to generatedifferent models over time, allowing for a comprehensive and intelligentcomparison of the structure between a first point in time (e.g.,pre-disaster) and a second point in time (e.g., post-disaster). In FIG.8A, first aerial imagery 830 of a building 820 in an initial(pre-disaster) condition 812 is collected and used to develop a firstthree-dimensional model (“first model”) 840 that can be rotated and/ormagnified to examine structural details. The building 820 depicted inthe first model 840 is a home that includes an undamaged roof 822,operation solar panels 824, and undamaged landscaping 826. The firstmodel 840 is stored in a database in association with the record for thedesignated property and can be accessed by an end-user to easily reviewand/or verify characteristics and attributes of the property.

In FIG. 8B, following a natural disaster, second aerial imagery 820 forthe building 820 in a damaged state 822 has been collected and used todevelop a first three-dimensional model (“second model”) 880 that can berotated and/or magnified to examine structural details. In other words,imagery obtained at a later time can be used to build a new (or updated)model of the building structure which is also stored in memory. In thisexample, the building 820 depicted in second model 880 can be seen tohave suffered extensive structural damage since the first aerial imagery830 was taken, here including a damaged roof 862, broken solar panels864, and yard debris 866, where a yard refers to an area of landimmediately adjacent to the building or a group of buildings, and may beeither enclosed or open.

In some embodiments, the proposed system can be configured toautomatically compare the initial or a previous 3D model and the new 3Dmodel. In such cases, a computing system may include software thatautomatically compares new 3D models of a building structure withprevious 3D models of the same building structure. The system couldperform predetermined comparisons to check for deviations in differentportions of the models. In one embodiment, a model comparisonapplication may be used to analyze the differences between two 3Dmodels. For example, the model comparison application could compare thedimensions of particular structural parts (such as walls, supportcolumns, doors, windows, etc.) and look for possible deviations. Also,the model comparison application could compare the relative orientationsof structural parts, including, for example, the angles between wallsand floors, walls and ceilings, and/or between parts of a building and aground surface. In other cases, the 3D models could be viewed by a userand manually compared.

For purposes of illustration, FIG. 9 depicts a situation in which two 3Dmodels are visually compared using a computing system. Specifically, theinitial model 840 of the building 820 is compared to the second model880 of the same building 820. In this case, second model 880 correspondsto a state of the building structure at a later time. Thus, the twomodels of the building 820 can be compared with one another to generatehighly accurate observations of the changes that have occurred duringthe period of time between the initial image capture and the subsequentimage capture. By reference to the two models, the system can beconfigured to automatically detect the differences in the appearance ofthe building 820 before and after the occurrence of the natural disasterand determine the specific areas of the building that have been damaged,as well as the extent of such damage. This assessment encompasses theentirety of the building by relying on a series of comprehensive imagerysets captured over multiple different angles and spatial views andacross different periods of time. It should be understood that while thefigures present two image capture sessions, in other embodiments, imagecapture sessions can occur at frequent intervals to allow for thecomparison of the damaged structure with the most up-to-date modelpreceding the disaster.

In FIG. 9, the two model inputs comprising first model 840 and secondmodel 880 are received by the system. A machine learning model, inconjunction with modeling software, is configured to determine thedifferences between the two models, as well as distinguish betweennormal changes to the property (e.g., additional trees and/orlandscaping, or the intentional removal of trees, updates in housecolor, roof type, new structural additions, etc.) and changes resultingfrom the natural disaster. In some embodiments, the detected deviationsmay be made precise using the modeling software which is able tocalculate substantially precise dimensions for parts of a structure, andalso determine angular measurements between different parts of surfaces.For example, a wall of a house may be normally disposed an angle ofapproximately 89 degrees with the ground surface. In contrast, followinga disaster, the same sidewall may make an angle of approximately 81degrees with the ground surface, indicating that the foundation on oneside of the building may be sinking or otherwise stressed/compressed.Thus, It may be appreciated that the present system may be useful fordetecting slight deviations that are not noticeable upon quick visualinspection and/or that are not perceptible from photographic images. Inthese cases, the utility of a system that can generate models of abuilding structure with sufficiently high precision can be clearly seen.Similarly, other components of interest, such as a tarp or debris, maybe based on the use of a trained material detection engine. The trainedmaterial detection engine may be trained to determine characteristics ofan asphalt shingle roof when it has full sun exposure, when it is in theshade, there is a cloud, or if there is damage to the roof, etc.

In this case, the system automatically identifies a first loss 910 and asecond loss 920 associated with portions of the roof, as well as a thirdloss 930 associated with a solar panel. Furthermore, a broken window hasbeen identified as a fourth loss 940, and a fallen tree as a fifth loss950. Additional damage may have occurred on the opposite side of thehouse and/or property, but for purposes of simplicity, only the front ofthe house is illustrated. For each loss identified, the system isconfigured to further determine the extent of the damage, as well asgenerate a repair estimate. Thus, the 3D models can be used to retrievethe exact dimensions for various portions of the building, providingsuch estimates with a high degree of accuracy. Furthermore, in someembodiments, the system may apply visually indicators to the areas wheredeviation (or damage) may have occurred or otherwise highlighting thelocation(s) of deviations that are suggestive of possible damage. Visualindicators may alert a user that further inspection of this part of thebuilding structure is needed. In some cases, this information could befurther used to make more accurate predictions about the value of a homefor insurance or resale purposes.

Furthermore, in some embodiments, a deep learning model can be used toestimate damages across different portions of the building. The inputsto this model are vertical (nadir) images and the outputs are a 2D “heatmap” of probable damage at different portions of the building. As anexample, an overview 1000 of one embodiment of the deep learning modelis illustrated with reference to FIG. 10. In FIG. 10, the deep learningmodel is being applied to a building (here, a residence 102) is shown. Atop-down view of a PIF property is presented in a first stage 1010,including a home 1050, a driveway with vehicles 1012, and a yard 1014.The property has experienced a natural disaster and as a result washeavily damaged.

In other words, in different embodiments, aerial imagery may be analyzedto determine if structure has sustained structural or roof damage. Forexample, image data may be analyzed to determine if a roof at a certaincoordinate has all of its corners and edges and whether said corners andedges of said roof appear to be damaged. In another example, image datais analyzed to determine the extent of damage that has occurred (e.g.,the percentage of damage to a component (e.g., roof) of structure, aswell as identify which portion of the structure is damaged (e.g., roof,walls, windows, porch, or personal property).

In a second stage 1020, the top-down or vertical imagery is input to thedeep learning model in order to generate bounded regions in which thedetected damage is located. In this case, a first damaged region 1022associated with the roof has been bounded, and a second damaged region1024 associated with debris in the yard has also been bounded. Thehigh-resolution imagery is then further processed to determine centroidsfor the damage and the probable extent of the damage as it radiatesoutward from the centroid. For example, a heat map or other a datavisualization technique may be generated that shows magnitude of damageassessed as color in two dimensions. The variation in color may be byhue or intensity, giving obvious visual cues to the reader about how thedamage is clustered or varies over space. Such representations offerend-users a mechanism by which to visualize complex data and understandit at a glance. In this example, a third stage 1030 and a fourth stage1040 depict a series of enclosed boundaries around each damaged regionthat serve as a visual indicator for the degree of damage. In otherwords, the system assigns different damage values or ranking andpresents this information in an easily consumable format similar toinfrared or heat maps, where the hue or intensity is more pronouncedtoward the portion of the structure where the damage was greatest. Inthis case, a first map 1032 and a second map 1042, each representingrich, complex data, have been generated and overlaid on the damagedregions. In another example, the pixel information from an image may beused to create a damage intensity plane that depicts an indication ofintensity of damage across a selected area.

The heat map may make use of image processing techniques to helpdetermine the extent of damage, such as filtering, edge detection (e.g.,Sobel operator, Prewitt operator, etc.), image hue, saturation,intensity, color, etc. thresholding, or binary image morphology. Imageprocessing techniques to help determine damage include informationassociated with trained machine learning engines, image classificationalgorithms, multi class classification, and anomaly detection. In someembodiments, these techniques may be used to correlate particular typesand levels of damage or material properties when a threshold is reached.Filtering and transform operators may enhance the image featuresindicative of damage. Thresholds applied to enhanced pixel values mayhelp to classify regions of damage for detection both spatially withinthe image and in terms of magnitude relative between regions within animage and relative between images. This may be a learned process wheredetermined thresholds are a function of information such as anticipateddamage type, lighting, time of day the picture was taken, or the sensorthat recorded the images. In different embodiments, learning (ortraining) may be implemented by common machine learning algorithms, suchas tensor flow, support vector machine, neural networks, autoencoders,Gaussian mixture models, or Naïve Bayes models, among others.

In different embodiments, insurance or other damage-related claims maybe proactively created based on the three-dimensional representation. Inanother example, the aerial vehicle may analyze data about a property(e.g., an image) to determine that additional data may be required andgenerate a request that further data be collected. In another example,the system may identify a problem associated with the structure andgenerate a message to alert insurers and/or homeowners about theproblem. For example, the system may identify that gas is leaking (e.g.,via visual detection of a problem with a pipe, via detecting the gas inthe air), and generate an alert to the end-user to shut off the gas.Similarly, such an approach may also be applied to leaking water or anyother problem that is detected to mitigate the possibility of additionaldamage being caused before other repairs can be initiated. In anotherexample, the system may initiate a remedial action, such as initiatingan insurance claim to address the damage to the property. Such a claimmay be initiated by contacting the insurance provider. In anotherexample, the system may be configured to initiate and adjust aninsurance claim at area without seeking prior authorization.

It is further contemplated herein that the analysis may includeconverting image data into a textual representation of at least some ofthat data, detecting whether power lines are active (e.g., via thermalimaging, via magnetic field detection), detecting whether a gas line isbroken (e.g., via thermal imaging to detect a leaking gas of a differenttemperature than the background, via analysis of the gas(es) in theair), or the like. Such analyses may be utilized to predict (and sendappropriate warnings) regarding possible future damages or accidentsthat may be caused by one or more of these conditions. In anotherexample, hyperspectral imagery is taken, utilized, and analyzed todetermine information such as what kind of damage was sustained, todetermine for example whether water damage was caused by salt water orfresh water. The type of water damage may be utilized to determine if aninsurance claim should be characterized as flood or storm damage or theextent of damage, such as damage to carpet, wood flooring, or the like.In another example, the aerial vehicle may gather thermal imagery, whichmay be utilized to identify hail (or other) damage to a home's roof.

FIG. 11 is a flow chart illustrating an embodiment of a method 1100 ofimproving the accuracy of a damage assessment based on aerial imagery. Afirst step 1110 includes obtaining a first set of imagery of a firstproperty captured at a first time, and a second step 1120 includesautomatically creating a first three-dimensional (3D) model of the firstproperty based on the first set of imagery. A third step 1130 includesobtaining a second set of imagery of the first property captured at asecond time subsequent to the first time, and a fourth step 1140includes automatically creating a second 3D model of the first propertybased on the second set of imagery. In addition, the method 1100includes a fifth step 1150 of automatically comparing the first 3D modeland the second 3D model using a machine learning model to detect damageon the first property that has occurred in the interval between thefirst time and the second time (e.g., where a natural disaster hasoccurred between those two time periods) and a sixth step 1160 ofautomatically generating and presenting a damage report based on thecomparison of the first 3D model and the second 3D model.

In other embodiments, the method may include additional steps oraspects. In one embodiment, the first set of imagery and the second setof imagery each include both nadir aerial images and oblique aerialimages for the first property. In another example in which the firstproperty includes a first structure, the method also includes steps ofautomatically employing a deep learning damage classification model todetermine that a first portion of the first structure has been damaged,automatically identifying varying damage magnitude levels associatedwith the first portion, and then automatically generating and presentinga first heat map that visually represents the damage magnitude levelsassociated with the first portion. In some other embodiments where thefirst property includes a yard, the method may also include steps ofautomatically employing a deep learning damage classification model todetermine that a first area of the yard includes debris, automaticallyidentifying varying damage magnitude levels associated with the firstarea, and automatically generating and presenting a first heat map thatvisually represents the damage magnitude levels associated with thefirst area.

In one example where the first property includes a first structure andeach of the first set of imagery and the second set of imagery includeoblique aerial images of the first structure, the method can alsoinclude automatically employing a machine learning model to evaluate theoblique aerial images and detect structural damage to sides of the firststructure. As another example, wherein the damage includes structuraldamage to a first portion of a structure located on the first property,the method can include automatically determining overall dimensions ofthe first structure based on the first 3D model and the second 3D model,automatically calculating, based on the overall dimensions of the firststructure, dimensions of the first portion, and automatically estimatingrepair costs for the first portion based on the calculated dimensions ofthe first portion. The estimate can be included in the report that isgenerated.

Other methods may be contemplated within the scope of the presentdisclosure. For example, in some embodiments, a method of improving theaccuracy of a damage assessment based on aerial imagery includes a firststep of obtaining a first set of imagery of a first property captured ata first time, and a second step of obtaining a second set of imagery ofthe first property captured at a second time subsequent to the firsttime, where a natural disaster has impacted the first property duringthe interval between the first time and the second time. In addition,the method includes a third step of feeding the first set of imagery andthe second set of imagery to a deep learning damage classificationmodel, and a fourth step of automatically determining, via the deeplearning classification model, that a first portion of a first structurelocated on the first property has been damaged, and automaticallyidentifying varying damage magnitude levels associated with the firstportion. Furthermore, the method includes a fifth step of automaticallygenerating and presenting a first heat map that visually represents thedamage magnitude levels associated with the first portion.

In other embodiments, this method may include additional steps oraspects. In one embodiment, the method may also include steps ofautomatically creating a first three-dimensional (3D) model of the firststructure based on the first set of imagery, automatically creating asecond 3D model of the first structure based on the second set ofimagery, automatically comparing the first 3D model and the second 3Dmodel using a machine learning model to detect damage on the firststructure that has occurred in the interval between the first time andthe second time, and automatically generating and presenting a damagereport based on the comparison of the first 3D model and the second 3Dmodel. In one embodiment, the first set of imagery and the second set ofimagery each include nadir aerial images for the first property.

In another example, the method can also include steps automaticallyemploying the deep learning damage classification model to determinethat a first area of a yard (located in the first property) includesdebris, and identifying varying damage magnitude levels associated withthe first area, and automatically generating and presenting a secondheat map that visually represents the damage magnitude levels associatedwith the first area. In some cases where the first set of imagery andthe second set of imagery each include oblique aerial images of thefirst structure, the method can also include automatically employing amachine learning model to evaluate the oblique aerial images and detectstructural damage to sides of the first structure.

In addition, in some embodiments, the method further includes steps ofautomatically determining overall dimensions of the first structurebased on the first 3D model and the second 3D model, automaticallycalculating, based on the overall dimensions of the first structure,dimensions of the first portion, and automatically estimating repaircosts for the first portion based on the calculated dimensions of thefirst portion. The estimate can be included in the report that isgenerated. In another example, the method may also include steps ofautomatically obtaining a third set of imagery of the first propertycaptured at a third time between the first time and the second time, andthen automatically updating the first three-dimensional (3D) model ofthe first structure based on the third set of imagery.

As described herein, the proposed systems and methods offer significantadvantages to damage assessment paradigms. Conventional approaches arelimited to crude classifications such as “damaged, “partially damaged”,or “not damaged” for the building as a whole, making estimations ofrebuilding costs difficult and/or unreliable. In contrast, the proposedsystem is designed to automatically assign different degrees of damageacross different portions of the building, allowing for more accurateestimates of the damage. For example, deep learning models are employedto identify condition concerns on the property, estimate rebuild costs,and other property attributes for each specific home with improvedaccuracy and specificity, unlike traditional probabilistic models thatuse features associated with such properties. These predictionaccuracies have been observed to greater than 98%. Furthermore, whileconventional approaches are limited to the detection of external damagebased on nadir imagery, the proposed systems are configured to useinformation from both nadir and oblique views to infer damage tointernal structures. In addition, while conventional approaches estimaterepair costs using simple models that do not account for the specificdimensions of a building or structure, the proposed system constructs 3Dmodels from aerial imagery that can be used to provide more accurateestimates of the spatial extent of the damage. Such 3D models can alsobe used to permit end-user to perform virtual inspections and damagereviews in lieu of ordering a costly physical inspection, facilitatinglarge scale damage assessments for each member in the catastrophe zone.Furthermore, the proposed system is readily scalable, and the code basefor data acquisition, processing, and UI presentation has been shown toprovide robust, reliable results.

The processes and methods of the embodiments described in this detaileddescription and shown in the figures can be implemented using any kindof computing system having one or more central processing units (CPUs)and/or graphics processing units (GPUs). The processes and methods ofthe embodiments could also be implemented using special purposecircuitry such as an application specific integrated circuit (ASIC). Theprocesses and methods of the embodiments may also be implemented oncomputing systems including read only memory (ROM) and/or random accessmemory (RAM), which may be connected to one or more processing units.Examples of computing systems and devices include, but are not limitedto: servers, cellular phones, smart phones, tablet computers, notebookcomputers, e-book readers, laptop or desktop computers, all-in-onecomputers, as well as various kinds of digital media players.

The processes and methods of the embodiments can be stored asinstructions and/or data on non-transitory computer-readable media. Thenon-transitory computer readable medium may include any suitablecomputer readable medium, such as a memory, such as RAM, ROM, flashmemory, or any other type of memory known in the art. In someembodiments, the non-transitory computer readable medium may include,for example, an electronic storage device, a magnetic storage device, anoptical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of suchdevices. More specific examples of the non-transitory computer readablemedium may include a portable computer diskette, a floppy disk, a harddisk, magnetic disks or tapes, a read-only memory (ROM), a random accessmemory (RAM), a static random access memory (SRAM), a portable compactdisc read-only memory (CD-ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), electrically erasable programmableread-only memories (EEPROM), a digital versatile disk (DVD and DVD-ROM),a memory stick, other kinds of solid state drives, and any suitablecombination of these exemplary media. A non-transitory computer readablemedium, as used herein, is not to be construed as being transitorysignals, such as radio waves or other freely propagating electromagneticwaves, electromagnetic waves propagating through a waveguide or othertransmission media (e.g., light pulses passing through a fiber-opticcable), or electrical signals transmitted through a wire.

Instructions stored on the non-transitory computer readable medium forcarrying out operations of the present invention may beinstruction-set-architecture (ISA) instructions, assembler instructions,machine instructions, machine dependent instructions, microcode,firmware instructions, configuration data for integrated circuitry,state-setting data, or source code or object code written in any of oneor more programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or suitable language, and proceduralprogramming languages, such as the “C” programming language or similarprogramming languages.

Aspects of the present disclosure are described in association withfigures illustrating flowcharts and/or block diagrams of methods,apparatus (systems), and computing products. It will be understood thateach block of the flowcharts and/or block diagrams can be implemented bycomputer readable instructions. The flowcharts and block diagrams in thefigures illustrate the architecture, functionality, and operation ofpossible implementations of various disclosed embodiments. Accordingly,each block in the flowchart or block diagrams may represent a module,segment, or portion of instructions. In some implementations, thefunctions set forth in the figures and claims may occur in analternative order than listed and/or illustrated.

The embodiments may utilize any kind of network for communicationbetween separate computing systems. A network can comprise anycombination of local area networks (LANs) and/or wide area networks(WANs), using both wired and wireless communication systems. A networkmay use various known communications technologies and/or protocols.Communication technologies can include, but are not limited to:Ethernet, 802.11, worldwide interoperability for microwave access(WiMAX), mobile broadband (such as CDMA, and LTE), digital subscriberline (DSL), cable internet access, satellite broadband, wireless ISP,fiber optic internet, as well as other wired and wireless technologies.Networking protocols used on a network may include transmission controlprotocol/Internet protocol (TCP/IP), multiprotocol label switching(MPLS), User Datagram Protocol (UDP), hypertext transport protocol(HTTP), hypertext transport protocol secure (HTTPS) and file transferprotocol (FTP) as well as other protocols.

Data exchanged over a network may be represented using technologiesand/or formats including hypertext markup language (HTML), extensiblemarkup language (XML), Atom, JavaScript Object Notation (JSON), YAML, aswell as other data exchange formats. In addition, informationtransferred over a network can be encrypted using conventionalencryption technologies such as secure sockets layer (SSL), transportlayer security (TLS), and Internet Protocol security (Ipsec).

While various embodiments of the invention have been described, thedescription is intended to be exemplary, rather than limiting, and itwill be apparent to those of ordinary skill in the art that many moreembodiments and implementations are possible that are within the scopeof the invention. Accordingly, the invention is not to be restrictedexcept in light of the attached claims and their equivalents. Also,various modifications and changes may be made within the scope of theattached claims.

We claim:
 1. A method of improving the accuracy of a damage assessmentbased on aerial imagery, the method comprising: obtaining a first set ofimagery of a first property captured at a first time; automaticallycreating a first three-dimensional (3D) model of the first propertybased on the first set of imagery, wherein the first set of imagery isclipped to generate first clipped images used to create the first 3Dmodel, the clipping of the first set of imagery comprising: querying ageographic information source for a first footprint polygon, identifyingfirst selected properties, from properties queried for in the first setof imagery, having a point in the first footprint polygon, re-geocodingthe first selected properties, and clipping images in the first set ofimagery at two zooms, based on the re-geocoding of the first selectedproperties; obtaining a second set of imagery of the first propertycaptured at a second time subsequent to the first time; automaticallycreating a second 3D model of the first property based on the second setof imagery, wherein the second set of imagery is clipped to generatesecond clipped images used to create the second 3D model, the clippingof the second set of imagery comprising: querying the geographicinformation source for a second footprint polygon, identifying secondselected properties, from properties queried for in the second set ofimagery, having a point in the second footprint polygon, re-geocodingthe second selected properties, and clipping images in the second set ofimagery at two zooms, based on the re-geocoding of the second selectedproperties; automatically comparing the first 3D model and the second 3Dmodel using a machine learning model to detect damage on the firstproperty that has occurred in the interval between the first time andthe second time; and generating and presenting a damage report based onthe comparison of the first 3D model and the second 3D model.
 2. Themethod of claim 1, wherein the two zooms for each set of imagery includea neighborhood level zoom and a structure level zoom.
 3. The method ofclaim 1, wherein the first property includes a first structure, and themethod further comprises: employing a deep learning damageclassification model to determine that a first portion of the firststructure has been damaged, and identifying varying damage magnitudelevels associated with the first portion; and generating and presentinga first heat map that visually represents the damage magnitude levelsassociated with the first portion.
 4. The method of claim 1, wherein thefirst property includes a yard, and the method further comprises:employing a deep learning damage classification model to determine thata first area of the yard includes debris, and identifying varying damagemagnitude levels associated with the first area; and generating andpresenting a first heat map that visually represents the damagemagnitude levels associated with the first area.
 5. The method of claim1, wherein the first property includes a first structure and each of thefirst set of imagery and the second set of imagery include obliqueaerial images of the first structure, and the method further comprisesemploying a machine learning model to evaluate the oblique aerial imagesand detect structural damage to sides of the first structure and tointerior portions of the first structure.
 6. The method of claim 1,wherein the damage includes structural damage to a first portion of astructure located on the first property, and the method furthercomprises: determining overall dimensions of the first structure basedon the first 3D model and the second 3D model; calculating, based on theoverall dimensions of the first structure, dimensions of the firstportion; and estimating repair costs for the first portion based on thecalculated dimensions of the first portion.
 7. A method of improving theaccuracy of a damage assessment based on aerial imagery, the methodcomprising: obtaining a first set of imagery of a first propertycaptured at a first time; clipping the first set of imagery to generatefirst clipped images, the clipping of the first set of imagerycomprising: querying a geographic information source for a firstcoverage polygon, selecting first selected properties, from propertiesqueried for in the first set of imagery, having a point in the firstcoverage polygon, re-geocoding the first selected properties, andclipping images in the first set of imagery at two zooms, based on there-geocoding of the first selected properties; obtaining a second set ofimagery of the first property captured at a second time subsequent tothe first time, wherein a natural disaster has impacted the firstproperty during the interval between the first time and the second time;clipping the second set of imagery to generate second clipped images,the clipping of the second set of imagery comprising: querying thegeographic information source for a second coverage polygon, selectingsecond selected properties, from properties queried for in the secondset of imagery, having a point in the second coverage polygon,re-geocoding the second selected properties, and clipping images in thesecond set of imagery at two zooms, based on the re-geocoding of theselected properties; feeding the first clipped images and the secondclipped images to a deep learning damage classification model;determining, via the deep learning classification model, that a firstportion of a first structure located on the first property has beendamaged, and identifying varying damage magnitude levels associated withthe first portion; and generating and presenting a first heat map thatvisually represents the damage magnitude levels associated with thefirst portion.
 8. The method of claim 7, further comprising:automatically creating a first three-dimensional (3D) model of the firststructure based on the first clipped images; automatically creating asecond 3D model of the first structure based on the second clippedimages; automatically comparing the first 3D model and the second 3Dmodel using a machine learning model to detect damage on the firststructure that has occurred in the interval between the first time andthe second time; and generating and presenting a damage report based onthe comparison of the first 3D model and the second 3D model.
 9. Themethod of claim 7, wherein the two zooms for each set of imagery includea neighborhood level zoom and a structure level zoom.
 10. The method ofclaim 7, wherein the first property includes a yard, and the methodfurther comprises: employing the deep learning damage classificationmodel to determine that a first area of the yard includes debris, andidentifying varying damage magnitude levels associated with the firstarea; and generating and presenting a second heat map that visuallyrepresents the damage magnitude levels associated with the first area.11. The method of claim 7, wherein the first set of imagery and thesecond set of imagery each include oblique aerial images of the firststructure, and the method further comprises employing a machine learningmodel to evaluate the oblique aerial images and detect structural damageto sides of the first structure and to interior portions of the firststructure.
 12. The method of claim 8, further comprising: determiningoverall dimensions of the first structure based on the first 3D modeland the second 3D model; calculating, based on the overall dimensions ofthe first structure, dimensions of the first portion; and estimatingrepair costs for the first portion based on the calculated dimensions ofthe first portion.
 13. The method of claim 8, further comprising:obtaining a third set of imagery of the first property captured at athird time between the first time and the second time; and automaticallyupdating the first three-dimensional (3D) model of the first structurebased on the third set of imagery.
 14. A system for improving theaccuracy of a damage assessment based on aerial imagery, the systemcomprising a processor and machine-readable media including instructionswhich, when executed by the processor, cause the processor to: obtain afirst set of imagery of a first property captured at a first time;automatically create a first three-dimensional (3D) model of the firstproperty based on the first set of imagery, wherein the first set ofimagery is clipped to generate first clipped images used to create thefirst 3D model, the clipping of the first set of imagery comprising:querying a geographic information source for a first footprint polygon,identifying first selected properties, from properties queried for inthe first set of imagery, having a point in the first footprint polygon,re-geocoding the first selected properties, and clipping images in thefirst set of imagery at two zooms, based on the re-geocoding of thefirst selected properties; obtain a second set of imagery of the firstproperty captured at a second time subsequent to the first time;automatically create a second 3D model of the first property based onthe second set of imagery, wherein the second set of imagery is clippedto generate second clipped images used to create the second 3D model,the clipping of the second set of imagery comprising: querying thegeographic information source for a second footprint polygon,identifying second selected properties, from properties queried for inthe second set of imagery, having a point in the second footprintpolygon, re-geocoding the second selected properties, and clippingimages in the second set of imagery at two zooms, based on there-geocoding of the second selected properties; automatically comparethe first 3D model and the second 3D model using a machine learningmodel to detect damage on the first property that has occurred in theinterval between the first time and the second time; and generate andpresent a damage report based on the comparison of the first 3D modeland the second 3D model, wherein the damage report comprises informationrelated to a probability of information in the damage report beingaccurate.
 15. The system of claim 14, wherein the two zooms for each setof imagery include a neighborhood level zoom and a structure level zoom.16. The system of claim 14, wherein the instructions further cause theprocessor to: employ a deep learning damage classification model todetermine that a first portion of a first structure located on the firstproperty has been damaged, and identifying varying damage magnitudelevels associated with the first portion; and generate and present afirst heat map that visually represents the damage magnitude levelsassociated with the first portion.
 17. The system of claim 14, whereinthe instructions further cause the processor to: employ a deep learningdamage classification model to determine that a first area of a yardlocated on the first property includes debris, and identify varyingdamage magnitude levels associated with the first area; and generate andpresent a first heat map that visually represents the damage magnitudelevels associated with the first area.
 18. The system of claim 14,wherein each of the first set of imagery and the second set of imageryinclude oblique aerial images of a first structure located on the firstproperty, and wherein the instructions further cause the processor toemploy a machine learning model to evaluate the oblique aerial imagesand detect structural damage to sides of the first structure and tointerior portions of the first structure.
 19. The system of claim 14,wherein the instructions further cause the processor to: determineoverall dimensions of a first structure located on the first propertybased on the first 3D model and the second 3D model; calculate, based onthe overall dimensions of the first structure, dimensions of the firstportion; and estimate repair costs for the first portion based on thecalculated dimensions of the first portion.
 20. The system of claim 14,wherein the instructions further cause the processor to: obtain a thirdset of imagery of the first property captured at a third time betweenthe first time and the second time; and automatically update the firstthree-dimensional (3D) model of the first structure based on the thirdset of imagery.